Time and ensemble averaging in time series analysis 
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In many applications expectation values are calculated by partitioning a single experimental time 
series into an ensemble of data segments of equal length. Such single trajectory ensemble (STE) is a 
counterpart to a multiple trajectory ensemble (MTE^ used whenever independent measurements or 
realizations of a stochastic process are available. The equivalence of STE and MTE for stationary 
systems was postulated by Wang and Uhlenbeck in their classic paper on Brownian motion (Rev. 
Mod. Phys. 17, 323 (1945)) but surprisingly has not yet been proved. Using the stationary 
and ergodic paradigm of statistical physics - the Ornstein-Uhlenbeck (OU) Langevin equation, we 
revisit Wang and Uhlenbeck's postulate. In particular, we find that the variance of the solution of 
this equation is different for these two ensembles. While the variance calculated using the MTE 
quantifies the spreading of independent trajectories originating from the same initial point, the 
variance for STE measures the spreading of two correlated random walkers. Thus, STE and MTE 
refer to two completely different dynamical processes. Guided by this interpretation, we introduce a 
novel algorithm of partitioning a single trajectory into a phenomenological ensemble, which we name 
a threshold trajectory ensemble (TTE), that for an ergodic system is equivalent to MTE. We find 
that in the cohort of healthy volunteers, the ratio of STE and TTE asymptotic variances of stage 
4 sleep electroencephalogram is equal to 1.96 ± 0.04 which is in agreement with the theoretically 
predicted value of 2. 

PACS numbers: 05.40.-a, 05.10.Gg, 05.45.Tp, 87.10.Mn 



The ergodic hypothesis asserting the equivalence of 
time and ensemble averages began with Boltzmann's |l| 
conjecture that a single trajectory can densely cover a 
surface of constant energy in phase space. His proof of 
the hypothesis as well as many subsequent proofs were 
shown to be fatally flawed. It was not until metric decom- 
posability was introduced by Birkoff [2] that a rigorous 
mathematical theory of ergodicity began to take shape. 
Kinchin, who wrote a seminal work on the mathemati- 
cal foundations of statistical mechanics Q , offered a sur- 
prisingly pragmatic approach to the ergodic hypothesis. 
He proposed to sidestep potentially formidable proofs of 
ergodicity and judge the theory constructed on such as- 
sumption by its practical success or failure. This latter 
perspective is adopted by the vast majority of physicists 
when averaging over independent measurements is not 
possible. 

Let us consider a stochastic process X{t) that can be 
the time series of the position of a particle undergoing the 
Brownian motion, inter-beat interval of human heart or a 
plethora of other time series generated by complex phys- 
ical or physiological systems. In the absence of direct 



evidence to the contrary, it is assumed that such pro- 
cesses are ergodic so that their underlying dynamics can 
in principle be deduced from a single, very often histor- 
ical, record X{t) measured over a sufhciently long time. 
In the classic paper Wang and Uhlenbeck wrote |4] : 

...One can then cut the record in pieces of length T 
(where T is long compared to all periods occurring in 
the process), and one may consider the different pieces 
as the different records of an ensemble of observations. 
In computing average values one has in general to distin- 
guish between an ensemble average and a time average. 
However, for a stationary process these two ways of av- 
eraging will always give the same result... 

This quote describes a ubiquitous process of generat- 
ing a phenomenological ensemble by partitioning a single 
dataset. In the context of time series analysis, we also use 
the synonym single trajectory ensemble (STE) as a coun- 
terpart to a multiple trajectory ensemble (MTE) which 
is employed whenever independent measurements or re- 
alizations of a stochastic process are available. Wang and 
Uhlenbeck have implicitly linked the of validity of phe- 
nomenological (STE) ensembles to the ergodicity of the 
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underlying dynamical system. 

It is remarkable that Wang and Uhlenbeck's postulate 
of equivalency of STE and MTE, which in fact provided 
justification of a half a century of empirical analyses, has 
not, to our knowledge, been directly tested. Motivated 
by our recent study [5| , we revisit this postulate using the 
stationary and ergodic paradigm of statistical physics - 
the Ornstein-Uhlenbeck (OU) Langevin equation: 



dX{t) 
dt 



-XXit)+ri{t) 
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where A is the dissipation rate [6]. In the above equa- 
tion, a zero-centered Gaussian random force rj (t) is delta 
correlated in time 
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and the angular brackets denote an average over an en- 
semble of realizations of the random force. X(t) is also 

im Q: 



a Gaussian random process with a spectrum 
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Consequently, from the convolution theorem one obtains 
the autocorrelation function of X{t) 
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X{t) may be expressed as the formal solution to the first- 
order, linear, stochastic differential equation ([T]) 



X{t) = e 



-\t 



X{0)+ / Tj{t')e^'' dt' 



(5) 



In Fig. [T] we present a solution X{t) generated by nu- 
merical integration of Eq. ([1]) with a constant time step 
At = l (A=0.025 and a^=7.8). 

According to Wang and Uhlenbeck's prescription, a 
long time series generated by the OU Langevin equation 
can be partitioned to yield a STE. We know that the 
solution to the OU Langevin equation is ergodic so that 
averages obtained using STE and MTE should coincide. 
Variance is the most commonly used measure of time se- 
ries variability. Therefore, let us perform computer sim- 
ulations to calculate this metric for both ensembles. In 
Fig. [21 cr^ of the solution X{t) of the Langevin equation 
^ is plotted as a function of the length t of the data 
window. The MTE variance cr^ is denoted by opened 
squares and was calculated using um — 3000 trajectories 
of length Nm = 1000 originating at zero. The STE vari- 
ance cr|, represented by open circles, was computed by 
partitioning a single trajectory of length Ns = umNm 
into segments of length r (we used a sliding window al- 
gorithm). It is obvious that the two ways of calculating 
the variance are not equivalent. 
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Figure 1: An example of the solution X{t) of the OU Langevin 
equation ^ for A=0.025 and a",,=7.8. The horizontal bars 
in the upper part of the figure represent those segments of 
the displayed trajectory whose left endpoints are equal, with 
a predetermined accuracy, to Xl- In other words, the left 
endpoints are the intersection of the trajectory with the cho- 
sen threshold which is marked in the graph by the horizontal 
gridline. Such segments of length t are used to construct a 
threshold trajectory ensemble discussed later in the text. 
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Figure 2: Variance of the solution X{t) of Eq. ([T]) is plotted 
as a function of the length of the data window r. The model's 
parameters are the same as in Fig. [T] The numerical estimate 
of variance was calculated using the single (circles), multiple 
(squares) and threshold (filled squares) trajectory ensembles. 
The theoretical value of the variance is drawn for the MTE 
(solid line, cf. Eq. (O) and the STE (dashed line, cf. Eq. 
(112^ 1. The sliding window algorithm was used to generate 
the STE. The non-overlapping window partitioning yielded 
the same results. The ratio of the asymptotic variances for 
STE and MTE is is given by Eq. JlS}. 



To clarify the difference between the two phenomeno- 
logical ensembles depicted in Fig. [5J we derive the ana- 
lytical expressions for the variance of the solution X{t) 
for both ensembles. Let us consider first the MTE of in- 
finite number of trajectories all starting from zero. Using 
Eq. ([5]) we obtain 



it) ^ {X- (t))^ {X {t))l = ^ [1 - . (6) 

Please note that STE averaging, by its very nature, in- 
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volves relative displacements: Z{t,T) = X{t + T) — X{t). 
Using Eq. (O, we can write X{t + t) as 



X{t + T) = e 



-At 



X{t) + J i]{t + t')e^''dt' 



and express Z(t, r) in the following form 



(7) 



r 

Zit, t) = e-^^ J T]{t + t')e^''dt' + X{t) [e-^" - l] . (8) 



Recall that the random force rj is stationary and, there- 
fore, the time translation of 77 in the integrand in Eq. 
does not affect the statistical properties of Z{t, r). With- 
out loss of generality, we may assume that a trajectory 
X{t) starts at zero and then Z{t, r) may be written in a 
particularly illuminating form 



Z{t,T) = X{T)+X{t) [e 
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For a fixed value of t (fixed left endpoint of the interval), 
the relative displacement Z[t, r) is a function of segment 
length T. In particular, the first term on the r.h.s. of 
Eq. ^ is stochastic while the second one is purely de- 
terministic. Consequently, the covariance of these two 
terms vanishes. Thus, the partitioning of a single tra- 
jectory is equivalent to building up deterministic trends. 
Z{t, t) being the sum of Gaussian variables is a Gaussian 
variable itself with zero mean value {E[Z{t, r)] = 0) and 
the following variance 

E[Z'{t,r)] = al,{r) + [e"^- - (10) 

The STE variance 0-5 (t) is just E[Z'^{t, r)] time averaged 
along the trajectory of length T 
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Taking into account that AT^l, we obtain the following 
approximation 
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Thus, the variance for the single trajectory ensemble is 
given by the formula Eq. © for the MTE, albeit with the 
effective dissipation rate Xeff — -^/2 which is half that 
of the MTE. Consequently, the ratio of the asymptotic 
variances for STE and MTE is 



0-5 (00) (00) 
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Both curves a1,j (r) and (t| (r) are plotted in Fig. [2] and 
are in agreement with the relevant numerical calcula- 
tions. 

In hindsight, the observed disagreement between the 
two ways of calculating the variance is less surprising 
than it ought to have been, ctm is the measure of spread- 
ing of statistically independent trajectories that start at 
X{0). On the other hand, the endpoints of intervals used 
to calculate the relative displacements Z{t, t) may be in- 
terpreted as the final positions of two correlated random 
walkers who both start at X(0) and whose correlation 
function p{t) has the exponential time dependence given 
by Eq. (j4|) . 0-5 (t) quantifies the spread of the distance 
between such walkers after time t and consequently refers 
to a completely different dynamical process. The formal 
proof of this interpretation is given elsewhere ■ 

Variance is certainly the most prevalent measure of 
time series variability. Moreover, it is often a critical 
part of fractal scaling detection algorithms, such as de- 
trended fluctuation analysis (DFA) [J, . In light of the 
difference between the STE and MTE variances, one can 
easily envision the situation when simultaneous applica- 
tion of both ensembles appears rational, but ultimately 
leads to systematic errors. For example, one may perform 
the measurements on a cohort of subjects to determine 
the variability of a physiological quantity. However, when 
the variability determined for a given patient is compared 
with that of the cohort, one may be inclined to improve 
the statistics by averaging over a single trajectory en- 
semble. We know that for the OU Langevin model, this 
approach leads to the gross overestimation of the asymp- 
totic variance. Thus, the question arises as to whether it 
is possible to partition a single trajectory in such a way 
that the resulting ensemble is equivalent to MTE and the 
improvement in statistics is accomplished. 

The solution to this problem presents itself as soon as 
we realize that in STE both endpoints of intervals con- 
tribute to spreading, whereas in MTE all left endpoints 
are the same initial condition. It is this difference be- 
tween the two ensembles that explains why the asymp- 
totic variance for STE is exactly twice that for MTE. 
The horizontal bars in the upper part of Fig. [T] represent 
those segments of the displayed trajectory whose left end- 
points are equal, with a predetermined accuracy e, to the 
Xl. In other words, the left endpoints are the intersec- 
tions of the trajectory with the chosen threshold which 
is marked in the graph by the horizontal gridline. Such 
segments of length r are used to construct a threshold 
trajectory ensemble (TTE). The filled squares in Fig. [2] 
correspond to the variance cr^ for such an ensemble. The 
data segments with Xl = (e = 0.87 which corresponds 
to 2.5% of the standard deviation of the trajectory of 
length Ns = umNm shown in Fig. [1]) were selected. 
There were approximately 60000 segments that satisfied 
the imposed criteria and, despite the relatively small size 
of the TTE, the agreement with the MTE is apparent. 
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Figure 3: The group averaged variance of stage 4 sleep elec- 
troencephalogram is plotted as a function of data window 
length r for STE (open circles) and TTE (filled squares). 
The thin solid lines represent the standard error of the mean. 
The sampling frequency of EEG was 200 Hz. The ratio of 
the asymptotic variances for STE and TTE 1.96 ± 0.04 is in 
agreement with the theoretical value given by Eq. (|13[1 . 

In Fig. [3] we present the group averaged variance of 
stage 4 sleep electroencephalogram (EEG) plotted as a 
function of data window length r for STE (open circles) 
and TTE (filled squares) . The retrospective analysis was 
performed on data from 10 healthy volunteers. For each 
subject, we extracted a single 5 minute data segment 
from the C3 channel of the polysomnogram e was 
chosen as 2.5% of the standard deviation asEG of EEG 
time series. For each volunteer, we averaged a variance 
curve (j'^{t) over the TTEs corresponding to the thresh- 
olds spaced by 2e between —2aEEG ^-nd 2aEEG- The 
single trajectory ensembles were constructed using the 
sliding window partitioning to account for the fact that 
the intervals making TTEs may overlap (c/. Fig. [T]). 
The comparison of Figs. [5] and [3] shows that both for the 
model and for experimental data the initial growth of 
variance is arrested. The OU Langevin models's param- 
eters were determined via a nonlinear fit of the group- 
averaged 0-2 to Eq. dni). The ratio of STE and TTE 
asymptotic variances for EEG is equal to 1.96 ± 0.04 
which is in agreement with the theoretically predicted 
value of 2 given by Eq. (|T3|) . The asymptotic values for 
the experimental data were obtained by averaging vari- 
ance for all intervals of length r > 1.5s. As we already 
mentioned, the value of theoretical ratio naturally stems 
from the two-walker interpretation of STE. The applica- 
tion of the OU Langevin equation to EEG modeling is 
described in detail in [5|. 

The equivalency of MTE and STE averages was pos- 
tulated as early as in 1945 by Wang and Uhlenbeck 
Herein we demonstrated the failure of this assumption 
for variance - the most frequently used measure of time 
series variability. By identifying the origin of the fail- 
ure, we were able to put forward a novel algorithm of 



partitioning a single trajectory into a threshold trajec- 
tory ensemble that for an ergodic system is equivalent to 
MTE. TTE obviates systematic errors which may be in- 
troduced into data analysis by simultaneous application 
of single and multiple trajectory ensembles. 

Let us finish with a cautionary note. The ergodic 
hypothesis dates back to the ve ry beg inning of statis- 
tical physics. The recent studies |llMl7| have once again 
brought ergodicity from the backstage into the lime- 
light. The growing list of non-ergodic systems should 
warn against indiscriminate application of single trajec- 
tory (single particle) ensembles. 
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